Skip to content

Conversation

@KSDaemon
Copy link
Member

@KSDaemon KSDaemon commented Nov 6, 2025

Native (rust) flatbuffer parser sometimes results in FlatBuffer parsing error for large datasets, while JS parser works.

The proble was because JS and Rust parsers use different FlatBuffer functions:

  1. JS Parser (packages/cubejs-cubestore-driver/codegen/http-message.ts:17-19):

    static getRootAsHttpMessage(bb:flatbuffers.ByteBuffer, obj?:HttpMessage):HttpMessage {
      return (obj || new HttpMessage()).__init(bb.readInt32(bb.position()) + bb.position(), bb);
    }
    • NO verification - just reads offset and initializes
    • ✅ Fast, no limits
  2. Rust Parser (rust/cubeorchestrator/src/query_message_parser.rs:51-52):

    let http_message = root_as_http_message(msg_data)
        .map_err(|_| ParseError::FlatBufferError)?;
    • Full verification with default limits
    • ❌ Fails on large datasets

Why It Fails?

FlatBuffers verifier has default limits:

  • max_tables - maximum number of tables (1M)

Client's data:

  • Each HttpRow + each HttpColumnValue = 1 table

This PR updates the native parser with overridden VerifierOptions with increased max_tables.

Check List

  • Tests have been run in packages where changes have been made, if available
  • Linter has been run for changed code
  • Tests for the changes have been added if not covered yet

@github-actions github-actions bot added rust Pull requests that update Rust code data source driver labels Nov 6, 2025
@codecov
Copy link

codecov bot commented Nov 6, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.11%. Comparing base (e8db92f) to head (302b16f).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #10121       +/-   ##
===========================================
+ Coverage   55.17%   78.11%   +22.94%     
===========================================
  Files         213      457      +244     
  Lines       16904    91016    +74112     
  Branches     3335     3335               
===========================================
+ Hits         9326    71093    +61767     
- Misses       7100    19445    +12345     
  Partials      478      478               
Flag Coverage Δ
cube-backend 55.17% <ø> (ø)
cubesql 83.34% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@KSDaemon KSDaemon marked this pull request as ready for review November 6, 2025 18:21
@KSDaemon KSDaemon requested review from a team as code owners November 6, 2025 18:21
Copy link
Member

@ovr ovr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with minor nits

@KSDaemon KSDaemon force-pushed the fix/flat-buffer-size branch from 3120ca9 to d04a455 Compare November 7, 2025 09:31
@KSDaemon KSDaemon force-pushed the fix/flat-buffer-size branch from d04a455 to 302b16f Compare November 7, 2025 11:22
@KSDaemon KSDaemon merged commit 5b048b2 into master Nov 7, 2025
85 of 86 checks passed
@KSDaemon KSDaemon deleted the fix/flat-buffer-size branch November 7, 2025 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data source driver rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants